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O . Abstract. Ill-posed inverse problems of the form y = Xp where y is J-dimensional vector of a data, p is m-dimensional 

probability vector which can not be measured directly and matrix X of observable variables is a known Jxra matrix, J < m, 
f"**^ ' are frequently solved by Shannon's entropy maximization (MaxEnt, ME). Several axiomatizations were proposed (see for 

instance [1], [2], [3], [4], [5], [6], [7], [8], as well as [9] for a critique of some of them) to justify the MaxEnt method (also) 
in this context. The main aim of the presented work is two-fold: 1) to view the concept of complementarity of MaxEnt and 
Maximum Likelihood (ML) tasks introduced at [10] from a geometric perspective, and consequently 2) to provide an intuitive 
and non-axiomatic answer to the 'Why MaxEnt?' question. 
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The concept of complementarity of maximum entropy and maximum likelihood tasks, proposed at [10], is in this 
vignette interpreted from a geometrical point of view of vectors. Two key notions are introduced: collinearity and 
coherence. In addition to shaping the complementarity into an elegant form the collinearity/coherence concepts offer 
<~| ■ an elementary answer to the persistent 'Why MaxEnt?' question. 



COLLINEARITY AND COHERENCE: ML AND MAXENT 

Definition 1. System of events {A±, A2, . . . ,A m } is equivalently described by its distribution p = \pi,P2, ■ ■ ■ ,p m ]> or 
by its potential u— [ui,u%,... ,U m ]- The relationship of equivalence is 

p = dist(u) = 1 - [e-«i , e-^ e—] 
Theorem 1. dist(u) = dist(u + C), where C is any constant. 

Note 1. The theorem implies that pmf is determined ( induced) uniquely by potential, and potential is determined by 
pmfup to an additive constant. 

Definition 2. Mean value of potential u weighted by pmfp is the scalar product of the vectors up. 
Definition 3. Two potential vectors a, ka, where k G K, are told to be collinear. 

Definition 4. Coherence of two pmfs p and q on potential u (or: with respect to potential u, or: relative to potential 
u) is defined as 

coher u(p,q) = up uq 
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(Zero value means the maximal possible coherence, the greater the (absolute) value, the smaller the coherence.) 
Definition 5. If coherence of two pmfs is zero, the pmfs are called coherent. 

Theorem 2. (ML task with simple exponential form) Let X be a random sample with vector of frequencies r. Let u 
be a potential. Then in class of distributions dist(Xu) induced by collinear with u potentials Au the most likely one 
dist(XQU) is coherent with r relative to potential u: 

coher u (r, dist(X uj) = 

Proof. For a proof see [10], proof of the Theorem 1. □ 

Definition 6. The mean value of potential u weighted by its own distribution is called entropy of the potential, 

eni(u) = udist(u) 

Theorem 3. (MaxEnt task with simple potential) Let X be a random sample with vector of frequencies r. Let u be a 
potential. Then in class of distributions dist(v) which are coherent with r relative to potential u 

coher u (r,dist(v)) = 

the most entropic one dist(v ) is induced by potential v = A u collinear with u. 

Proof. For a proof see [10], proof of the Theorem 1. □ 




FIGURE 1. Geometric representation of Theorem 2, 3 

From a geometrical standpoint, claims of the Theorem 1 and 2 lead to a search for an intersection of u with a line 
defined by an orthogonal to u vector r — q, as it is depicted on the Figure 1 . 

The circular relationship of the claims of Theorem 2 and Theorem 3, dubbed in [10] 'complementarity of Maximum 
Likelihood and MaxEnt tasks', is visualized by the following diagram: 

collin > ML 

I I 

ME < coher 

In an intentionally loose manner the complementarity can be stated as: 'in the class of collinear potentials the most 
probable (the most likely) is the coherent one; in the class of coherent distributions the most entropic is the collinear 
one'. 

In light of the MaxProb rationale of MaxEnt (see [11]), the above statement can achieve an even deeper symmetry, 
in the case of sufficiently large sample. Then the words 'the most entropic' can be replaced by 'the most probable', 
and the diagram can be redrawn: 
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WHY MAXENT? A SIMPLE ANSWER 

Theorem 3 offers a simple argument in favor of Shannon's entropy maximization (MaxEnt) method/criterion in the 
inverse-problem context, which was mentioned at the abstract. 

Consider the following 'game' which models the inverse problem: an experiment reveals m different outcomes. The 
experiment was repeated sufficiently many times and vector of frequencies of the outcomes in the obtained random 
sample r = [ri,r2, . . . ,r m ] is known to us. Also, we are told a potential u. Given this information {r,u} we are 
asked to pick up a potential vo of a set of all potentials v whose distributions are coherent with r on potential u (in 
other words: the set consists of all such potentials v that ru = dist(v)u). Common sense dictates to choose vo = u, 
but since it is unlikely that this choice will satisfy the coherence condition, the second simplest possible choice is a 
collinear with u potential v = Au where A should be chosen such that dist(Xu) will attain coherence with r on the 
potential u. 

The fact that MaxEnt (and to the best of our knowledge no other method) chooses in the above 'game' just the 
collinear with u potential, hence the simplest possible solution, can be used as an answer to 'Why MaxEnt?' question. 

In order to make relationship of the above mentioned 'game' to the ill-posed inverse problem clear, note that the pair 
of information {r, u} forms the value of y, as y = ru, and the ill-posed problem becomes y = up. This way obviously 
extends also to more dimensional y, hence to the case where more potentials are given. 
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Note. As compared to the paper which has appeared at the MaxEnt proceedings, here Theorems 2 and 3 were 
re-stated in terms of random sample, so that the argument made at the last Section should be easier to grasp. 
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